The invention relates to the field of vision guided robotics, and more particularly to a method and apparatus for single camera three dimensional vision guided robotics.
Robots have long been widely used in manufacturing processes for many applications. Many different types of sensors are used to guide the robot but machine vision is increasingly being used to guide robots in their tasks. Typically such machine vision is used in a two-dimensional application wherein the target object need only be located in an x-y plane, using a single camera. For example see U.S. Pat. No. 4,437,114 LaRussa. However many robotic applications require the robot to locate and manipulate the target in three dimensions. In the past this has involved using two or more cameras. For example see U.S. Pat. No. 4,146,924 Birk et al.; and U.S. Pat. No. 5,959,425 Bieman et al. In order to reduce hardware costs it is preferable to use a single camera. Prior single camera systems however have used laser triangulation which involves specialized sensors, must be rigidly packaged to maintain geometric relationships, require sophisticated inter-tool calibration methods and tend to be susceptible to damage or misalignment when operating in industrial environments.
Target points on the object have also been used to assist in determining the location in space of the target object using single or multiple cameras. See U.S. Pat. Nos. 4,219,847 Pinkney et al. and U.S. Pat. Nos. 5,696,673; 5,956,417; 6,044,183 and 6,301,763 all of Pryor. Typically these methods involve computing the position of the object relative to a previous position, which requires knowledge of the 3D location of the object at the starting point. These methods also tend to not provide the accuracy and repeatability required by industrial applications. There is therefore a need for a method for calculating the 3D position of objects using only standard video camera equipment that is capable of providing the level of accuracy and repeatability required for vision guidance of robots as well as other applications requiring 3D positional information of objects.
A method for three-dimensional handling of an object by a robot using a tool and one camera mounted on the robot end-effector is disclosed in which object features or landmarks are used to calculate the three-dimensional pose of the object. The process is performed in three main steps:
a) calibration of the camera;
b) selecting of the features on the object;
c) finding the three-dimensional pose of the object and using this information to guide the robot to the object to perform any operations (e.g. handling, cutting, etc.).
According to one aspect of the invention, the invention provides a method of three-dimensional handling or an object by a robot using a tool and one camera mounted on the robot. The method involves first calibrating the camera by finding a) the camera intrinsic parameters; b) the position of the camera relative to the tool of the robot (xe2x80x9chand-eyexe2x80x9d calibration); and c) the position of the camera in a space rigid to the place where the object will be trained (xe2x80x9cTraining Spacexe2x80x9d). Next the object features are taught by a) putting the object in the xe2x80x9cTraining Spacexe2x80x9d and capturing an image of the object with the robot in the calibration position where the xe2x80x9cCamera to Training Spacexe2x80x9d transformation was calculated; b) selecting at least 6 visible features from the image; c) calculating the 3D position of each feature in xe2x80x9cTraining Spacexe2x80x9d; d) defining an xe2x80x9cObject Spacexe2x80x9d aligned with the xe2x80x9cTraining Spacexe2x80x9d but connected to the object and transposing the 3D coordinates of the features into the xe2x80x9cObject Spacexe2x80x9d; c) computing the xe2x80x9cObject Space to Cameraxe2x80x9d transformation using the 3D position of the features inside the xe2x80x9cObject Spacexe2x80x9d and the positions of the features in the image; f) defining an xe2x80x9cObject Framexe2x80x9d inside xe2x80x9cObject Spacexe2x80x9d to be used for teaching the intended operation path; g) computing the Object Frame position and orientation in xe2x80x9cTool Framexe2x80x9d using the transformation from, xe2x80x9cObject Frame to Cameraxe2x80x9d and xe2x80x9cCamera to Toolxe2x80x9d; h) sending the xe2x80x9cObject Framexe2x80x9d to the robot; and i) training the intended operation path relative to the xe2x80x9cObject Framexe2x80x9d using the robot. Next the object finding and positioning is carried out by a) positioning the robot in a predefined position above the bin containing the object and capturing an image of the object; b) if an insufficient number of selected features are in the field of view, moving the robot until at least 6 features can be located; c) with the positions of features from the image and their corresponding positions in xe2x80x9cObject Spacexe2x80x9d as calculated in the training step, computing the object location as the transformation between the xe2x80x9cObject Spacexe2x80x9d and xe2x80x9cCamera Space xe2x80x9d; d) using the transformation to calculate the movement of the robot to position the camera so that it appears orthogonal to the object; e) moving the robot to the position calculated in step d); f) finding the xe2x80x9cObject Space to Camera Spacexe2x80x9d transformation in the same way as in step c); g) computing the object frame memorized at training using the found transformation and xe2x80x9cCamera to Toolxe2x80x9d transformation; h) sending the completed xe2x80x9cObject Framexe2x80x9d to the robot; and i) using the xe2x80x9cToolxe2x80x9d position to define the frame in xe2x80x9cRobot Spacexe2x80x9d and performing the intended operation path on the object inside the xe2x80x9cRobot Spacexe2x80x9d.
According to a further aspect of the invention, there is provided a method of three-dimensional bridling of an object by a robot rising a tool and one camera mounted on the robot. The method involves first calibrating the camera by finding a) the camera intrinsic parameters; and b) the position of the camera relative to the tool of the robot (xe2x80x9chand-eyexe2x80x9d calibration). Next the object features are taught by a) putting the object in the field of view of the camera and capturing an image of the object; b) selecting at least 6 visible features from the image; c) calculating the 3D position in real world co-ordinates of the selected features inside a space connected to the object (xe2x80x9cObject Spacexe2x80x9d); d) computing the xe2x80x9cObject Space to Cameraxe2x80x9d transformation using the 3D position of the features inside this space and the position in the image; e) defining an xe2x80x9cObject Framexe2x80x9d inside xe2x80x9cObject Spacexe2x80x9d to be used for teaching the handling path; f) computing the xe2x80x9cObject Framexe2x80x9d position and orientation in xe2x80x9cTool Framexe2x80x9d using the transformation from xe2x80x9cObject Frame to Cameraxe2x80x9d and xe2x80x9cCamera to Toolxe2x80x9d; g) sending the computed xe2x80x9cObject Framexe2x80x9d to the robot; and h) training the intended operation path inside the xe2x80x9cObject Framexe2x80x9d. Next the object finding and positioning is carried out by a) positioning the robot in a predefined position above the bin containing the target object; b) if an insufficient number of selected features are in the field of view, moving the robot until at least 6 features can be located; c) with the positions of features from the image and their corresponding position in xe2x80x9cObject Spacexe2x80x9d as calculated in the training session, computing the object location as the transformation between the xe2x80x9cObject Spacexe2x80x9d and xe2x80x9cCamera Spacexe2x80x9d; d) using the said transformation to calculate the movement of the robot to position the camera so that it appears orthogonal to the object; e) finding the xe2x80x9cObject Space to Camera Spacexe2x80x9d transformation in the same way as in step d); f) computing the object frame memorized at training using the found transformation and xe2x80x9cCamera to Toolxe2x80x9d transformation; g) sending the computed xe2x80x9cObject Framexe2x80x9d to the robot; and h) using the xe2x80x9cToolxe2x80x9d position to define the frame in xe2x80x9cRobot Spacexe2x80x9d and performing the intended operation path on the object inside the xe2x80x9cRobot Spacexe2x80x9d. The invention also provides a system for carrying out the foregoing methods.